An Examination of Variation in Rater Severity Over Time: A Study in Rater Drift

Ratings on performance tasks from two scoring sessions of an eighth grade mathematics examination, developed by the California State Department of Education, were used (a) to study the feasibility of estimating IRT rater severity information within a scoring session, (b) to investigate the variation in rater severity within rating sessions (which we called rater drift), and (c) to examine the relationship (or lack thereof) of the IRT rater severity information to traditional information based on expert re-ratings. We found that it was indeed feasible to give feedback on half-day intervals, that that information was interpretable by the raters, that there was considerable variation in rater severity within the scoring session, and that the IRT information was not redundant with the traditional information. We also investigated the impact of the estimated rater severities using an IRT approach, and showed that they accounted for about half of the residual misfit. In addition, we found that table leaders (expert raters assigned to lead small groups of raters) exhibited more variation in rater severity than the regular raters. This observation may have as much to do with the somewhat odd sample of student work that they examine as it does to any systematic pattern of ratings on their part. We investigated the effect of table leader’s severity on the raters they were leading, and found that there was no systematic